Papers with compact representations
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024.findings-eacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | Embeddings in transformers encode vast amounts of linguistic nuances and patterns. |
| Approach: | They investigate the anisotropy dynamics and intrinsic dimension of embeddings in transformers . they found that transformer decoders exhibit a bell-shaped anisotropie profile . |
| Outcome: | The investigated embeddings exhibit a bell-shaped curve with the highest anisotropy concentrations in the middle layers . the intrinsic dimension increases in the initial phases of training, indicating an expansion into higher-dimensional space. |
Multi-turn Response Selection using Dialogue Dependency Relations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-turn response selection ignore the dependencies between the turns. |
| Approach: | They propose a dialogue extraction algorithm to transform a dialog history into threads based on their dependency relations. |
| Outcome: | The proposed model outperforms the state-of-the-art models on DSTC7 and DSTF8* with competitive results on UbuntuV2 . |
LightThinker: Thinking Step-by-Step Compression (2025.emnlp-main)
Copied to clipboard
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models have demonstrated their remarkable capabilities in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. |
| Approach: | They propose a method that dynamically compresses verbose thought steps into compact representations and discards original reasoning chains. |
| Outcome: | The proposed method reduces peak memory usage and inference time while maintaining competitive accuracy. |
Enhancing Lexicon-Based Text Embeddings with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. |
| Approach: | They introduce the first lexicon-based embeddings that consolidates the vocabulary space through token embeddation clustering to handle the issue of token redundancy in LLM vocabularies. |
| Outcome: | The proposed model outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) it also supports efficient dimension pruning without any specialized objectives like Matryoshka Representation Learning. |